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SUMMARY 

Symbology  to  represent  fixed-wing  and  rotary-wing  Unmanned  Aerial  Vehicles  (UAVs)  was 
developed  and  compared  to  MIL-STD-1787C  symbology.  The  symbology  was  evaluated  in 
several  simulated  UAV  operator  tasks.  In  the  first  portion  of  the  study,  static  screen-shots  from  a 
simulated  UAV  control  station  display  screen  were  presented  to  participants.  Participants  were 
required  to  determine  how  many  air  vehicle  symbols  of  a  particular  type  or  affiliation  were 
present  in  the  screen-shot.  The  results  indicated  that  the  new  symbology  was  recognized  at  least 
as  quickly  and  accurately  as  existing  MIL-STD-1787C  symbology.  In  the  second  portion  of  the 
study,  participants  monitored  dynamic  video  on  the  control  station  display.  Participants  were 
required  to  monitor  the  display  for  specific  events  associated  with  their  ownships.  Again, 
performance  with  the  new  symbology  was  at  least  as  good  as  that  with  MIL-STD-1787C 
symbology.  While  this  study  was  limited  in  scope,  the  results  represent  a  productive  step  in  the 
development,  evaluation,  and  standardization  of  new  UAV  symbology. 
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INTRODUCTION 

The  function  of  a  symbolic  representation  is  to  convey  infonnation  about  a  referent  subject 
or  situation.  When  selecting  or  developing  a  standard  symbology  set  for  use  in  the  command  and 
control  of  unmanned  aerial  systems  (UASs),  a  driving  factor  in  the  selection  or  development 
process  should  be  the  degree  to  which  the  symbology  set  allows  an  operator  to  rapidly 
understand  the  referent  subject  or  situation  from  the  symbolic  representations  and  maintain 
situation  awareness. 

In  developing  a  tactical  display  symbology  set  for  UASs,  one  approach  is  to  incorporate 
new  UAS  symbology  into  existing  military  standards.  Two  military  standard  (MIL-STD) 
symbology  sets  that  represent  air  vehicle  operations  are  MIL-STD- 1878C  and  MIL-STD-2525B. 
MIL-STD-1787C  details  the  symbolic  representations  to  be  used  in  aircraft  cockpit  interfaces. 
MIL-STD-2525B  is  a  larger  and  much  more  general  symbology  set  designed  to  function  as  a 
universal  C4I  class  of  symbols.  As  such,  the  focus  of  MIL-STD-2525B  on  aerial  vehicles  is 
limited  to  a  symbolic  representation  of  air  vehicles  within  a  larger  battlespace  environment.  No 
“inside  the  cockpit”  operations  are  captured  within  MIL-STD-2525B. 

It  is  not  entirely  clear  which  standard  has  precedence  for  UAS  operations.  MIL-STD- 
2525B  states  that  it  “applies  to  all  DOD  components  directly  or  indirectly  involved  with  C4I 
operations,  system  operations,  system  development,  and  training  within  the  context  of 
warfighting  operations,  and  that  MIL-STD-2525B  will  serve  as  the  standard  symbol  set  for  all 
future  DOD  uses  of  C4I  symbology.  The  standard  contains  symbology  that  can  be  applied  to 
mapping/charting,  weather,  cockpit  display,  and  engineering  design  symbology  to  the  extent  that 
it  is  usable  by  these  communities.  The  standard  will  apply  to  all  future  use  of  symbols  in  two 
dimensional  and  electronic  display  systems  in  C4I  environments.”  MIL-STD-2525  also  states 
that  “MIL-STD-1787C-Aircraft  Display  Symbology  has  been  developed  to  provide  standards 
guidance  regarding  rotary-wing  and  fixed-wing  cockpit  displays.”  Furthermore,  as  noted  in  MIL- 
STD-1787C,  “The  tactical  display  symbology  in  this  standard  is  to  be  used  as  an  alternative  to 
the  symbology  in  MIL-STD-2525,  which  is  not  appropriate  for  aircraft  use.”  While  the 
applicability  of  MIL-STD-1787C  and  MIL-STD-2525B  suggests  that  MIL-STD-1787C  may  take 
precedence  in  aircraft  cockpit  display,  it  is  not  clear  which  document  takes  precedence  for  UAS 
control  systems. 

The  choice  between  utilizing  MIL-STD-1787C  or  MIL-STD-2525B  as  the  symbology  set 
for  UASs  is  complicated  by  the  variety  of  levels  of  control  found  in  UASs,  from  man-in-the-loop 
remotely  piloted  vehicle  type  operations  to  completely  or  almost  completely  autonomous  UAS 
operations.  While  MIL-STD-1787C  holds  obvious  advantages  for  man-in-the-loop  piloting 
operations,  MIL-STD-2525B  holds  advantages  for  a  UAS  operator  engaged  primarily  in 
information  management  other  than  that  associated  with  direct  piloting  of  an  air  vehicle. 
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It  should  be  noted  that  there  are  limitations  associated  with  both  MIL-STD-1787C  and 
MIL-STD-2525B  in  the  context  of  UAS  operations.  Neither  symbology  set  deals  with  a  number 
of  factors  and  concerns  for  UAS  operations.  For  example,  neither  standard  deals  with  the 
symbolic  representation  of  multiple  air  vehicles  under  the  control  of  a  single  operator.  Given  the 
limitations  of  both  standards,  the  final  symbology  set  for  UAS  operations  will  necessarily 
involve  a  significant  modification  of  either  MIL-STD-1787C  or  MIL-STD-2525B. 


In  evaluating  the  relative  merits  of  MIL-STD-1787C  versus  MIL-STD-2525B,  and  in  the 
development  of  new  UAS  symbologies,  several  issues  should  be  considered  including  situation 
awareness,  recognition  speed,  and  errors  associated  with  processing  information.  In  the  present 
study,  a  set  of  UAS  symbols  was  developed  for  evaluation  in  the  context  of  MIL-STD-1787C 
symbology.  Symbols  for  both  fixed-wing  and  rotary-wing  unmanned  air  vehicles  (UAVs)  were 
created  (see  figure  1).  The  symbols  were  developed  in  accordance  with  the  standard  human 
factors  criteria  that  the  symbols  must  allow  one  to  rapidly  recognize  the  symbols  and  accurately 
distinguish  the  symbols  from  other  related  symbology.  The  symbols  were  then  utilized  and 
evaluated  in  several  simulated  UAS  operator  tasks.  The  aim  of  this  study  was  to  evaluate  the 
degree  to  which  these  newly  developed  symbols  could  be  effectively  utilized  within  a  larger 
MIL-STD-1787C  symbology  set  for  UAS  operations. 


Figure  1 :  Examples  of  new  UAV  symbology  developed  for  this  study. 

(a  -  Rotary- wing  UAV,  unknown  affiliation;  b  -  Fixed-wing  UAV,  ownship) 

METHOD 


PARTICIPANTS 


Twenty  male  and  female  university  students  between  the  ages  of  18  and  39  served  as 
participants.  The  average  age  was  24.  The  study  was  approved  by  the  institutional  review  board 
at  East  Central  University,  located  near  Ada,  Oklahoma  (approximately  65  miles  southeast  of 
Oklahoma  City).  All  participants  signed  an  informed  consent  statement  prior  to  participating  in 
the  study. 
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TRAINING 


Each  participant  was  first  given  approximately  45  min  of  individual  instruction  and  training 
prior  to  starting  the  evaluation.  Training  consisted  of  a  tutorial  session  in  which  participants  were 
given  an  explanation  of  UAVs,  control  stations,  symbology,  and  air  operations.  Participants 
viewed  a  chart  depicting  MIL-STD-1787C  symbology  and  the  symbology  developed  for  this 
study,  static  screen-shots  from  Open  Unmanned  Mission  Interface  (Open  UMI)  scenarios  (see 
figure  2),  and  several  videos  of  Open  UMI  scenarios.  Open  UMI  is  an  existing  vehicle  control 
interface  that  allows  for  the  simultaneous  monitoring  and  control  of  multiple  unmanned  vehicles. 
Participants  were  then  given  an  opportunity  to  review  and  study  the  symbology  set  in  order  to 
commit  it  to  memory.  The  researcher  provided  clarification  and  explanation  whenever 
participants  requested  it.  At  the  completion  of  the  training  session,  participants  were  given  a  brief 
test  in  order  to  ensure  that  they  understood  the  symbology  set  and  the  Open  UMI  display. 
Participants  were  required  to  correctly  identify  90%  of  symbols  before  proceeding  to  the 
evaluation  phase.  All  participants  reached  the  criterion  of  90%  accuracy  after  their  first  training 
session. 


Figure  2:  Example  of  an  Open  UMI  scenario  showing  UAV  symbology  developed  for  this 
study,  and  MIL-STD-2525B  symbology  substituted  for  MIL-STD-1787C  symbology.  A  copy  of 
MIL-STD-1787C  can  be  requested  from  ASC/ENOI,  Wright-Patterson  AFB,  Ohio,  45433-7101. 
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EVALUATION 


The  quantitative  human  factors  evaluation  process  entailed  participants  conducting  rapid 
evaluations  of  events  from  the  Open  UMI  tactical  display.  These  static  scenes  and  prerecorded 
video  events  involved  UAS  and  non-UAS  vehicles  in  varying  levels  of  density  and  distribution 
about  the  display.  Researchers  collected  data  from  participants  in  the  form  of  latencies  to 
perform  recognition/identification  tasks,  errors  in  processing  information  about  the  scenarios, 
and  the  degree  to  which  participants  were  able  to  develop  and  maintain  situation  awareness  of 
the  scenarios  being  presented.  For  each  analysis,  participants  were  presented  with  scenarios  at 
three  levels  of  air  vehicle  density,  allowing  an  analysis  of  performance  within  sparsely, 
moderately,  and  heavily  populated  Open  UMI  scenarios. 

RECOGNITION  SPEED 


During  the  Recognition  Speed  phase  of  the  evaluation,  participants  were  presented  with 
static  screen-shots  from  the  Open  UMI  control  station  and  were  required  to  rapidly  determine 
how  many  air  vehicle  symbols  of  a  particular  type  or  affiliation  were  present.  For  instance, 
participants  were  asked  to  rapidly  determine  how  many  bombers  were  present.  The  participants 
would  then  carry  out  this  task  on  nine  separate  screen-shots.  When  completed,  the  participant 
would  then  move  on  to  another  class  of  air  vehicles  such  as  fighters  or  fixed-wing  UAVs. 
Screen-shots  varied  in  how  densely  populated  the  symbols  were.  Participants  performed  this  task 
on  nine  screen-shots  for  each  of  the  symbol  categories  listed  below: 

1 .  Friend 

2.  Foe 

3.  Neutral 

4.  Unknown  affiliation 

5.  Ownship 

6.  Bomber 

7.  Helicopter 

8.  Cruise  missile 

9.  Heavy 

10.  Fighter  low  tech 

1 1 .  Fighter  high  tech 

12.  F  ixed- wing  UA  V 

1 3 .  Rotary- wing  UAV 

14.  Unknown  class 


Thus,  participants  performed  this  task  for  126  total  screen-shots.  Latencies  to  respond  were 
recorded  in  milliseconds.  The  number  of  errors  per  screen-shot  was  also  calculated. 
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RESULTS 


LATENCIES 


The  results  indicate  that  the  latencies  to  recognize  the  new  UAS  symbols  fell  within  the 
range  of  latencies  to  recognize  symbols  from  MIL-STD-1787C  (see  table  1).  That  is,  participants 
were  as  fast  with  the  new  UAS  symbols  as  they  were  with  the  MIL-STD-1787C  symbols. 
Latencies  associated  with  affiliation  (M=  1.670)  were  shorter  than  latencies  for  vehicle  class  (M 
=  3.592;  f(235)  =  19.93,  p  <  .001).  This  is  almost  certainly  due  to  the  fact  that  color  variations 
associated  with  affiliation  symbology  are  much  more  salient  attributes  than  complex  shape 
variations  associated  with  vehicle  class  symbology.  We  also  compared  whether  the  new 
“ownship”  color  (blue)  was  recognized  as  quickly  as  the  existing  MIL-STD-1787C  affiliation 
colors.  The  latency  to  recognize  ownship  symbology  (M  =  1.685)  was  not  statistically  different 
than  the  latency  to  recognize  other  affiliations  (M=  1.622;  /(34)  =  0.12,  ns).  There  was  also  no 
statistically  significant  difference  in  the  latencies  to  recognize  the  new  UAV  rotary-wing 
symbology  (M=  3.849)  compared  to  MIL-STD-1787C  helicopter  symbology  (M=  3.792;  t( 38)  = 
0.20,  ns).  Additionally,  there  was  no  difference  in  the  latencies  to  recognize  the  new  UAV  fixed- 
wing  symbology  (M=  4.099)  compared  to  all  other  MIL-STD-1787C  fixed-wing  symbology  (M 
=  3.719;  t(53)=  1.70,  ns). 


Table  1:  Latencies  to  Recognize  Symbology 


Latency  (sec 

Foe 

1.459 

Unknown 

1.587 

Neutral 

1.601 

Ownship 

1.685 

Friendly 

2.035 

Unknown  class 

2.582 

Bomber 

2.993 

Heavy 

3.315 

Cruise  missile 

3.561 

Helicopter 

3.792 

UAV  rotary-wing 

3.849 

Low  tech  fighter 

4.018 

UAV  fixed-wing 

4.099 

High  tech  fighter 

4.528 

ERROR  ANALYSIS 


Errors  included  the  failure  to  recognize  symbols  or  the  misidentification  of  symbols.  The 
results  indicate  that  the  number  of  errors  with  the  new  UAS  symbols  fell  within  the  range  of 
errors  found  with  symbols  from  MIL-STD-1787C.  That  is,  the  number  of  errors  associated  with 
the  new  UAS  symbols  did  not  differ  substantially  from  the  number  of  errors  associated  with  the 
MIL-STD-1787C  symbols  (see  table  2).  The  number  of  errors  for  affiliation  (M  =  .077)  was 
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significantly  less  than  the  number  of  errors  vehicle  class  (M=  .150;  1(270)  =  4.64 ,p  <  .001).  The 
number  of  errors  associated  with  the  new  ownship  color  (M  =  .062)  was  not  significantly 
different  from  the  number  of  errors  seen  with  colors  associated  with  existing  MIL-STD-1787C 
affiliation  symbology  (M=  .080;  1(27)  =  0.72,  ns).  The  number  of  errors  associated  with  the  new 
UAV  fixed-wing  symbol  (M=  .170)  was  not  significantly  different  from  those  seen  with  existing 
MIL-STD-1787C  fixed-wing  symbols  (M  =.148;  1(21)  =  1.02,  ns).  The  number  of  errors 
associated  with  the  new  UAV  rotary-wing  symbol  (M=  .156)  was  significantly  smaller  than  that 
associated  with  the  existing  MIL-STD-1787C  helicopter  symbol  (M=  .289;  1(38)  =  2.55 ,p  <.05). 
For  some  symbols,  the  error  rates  were  rather  high.  For  example,  the  symbol  for  heavy  aircraft 
had  an  error  rate  of  0.211,  and  that  for  helicopters  was  0.289.  One  possible  explanation  is  that 
participants  were  encouraged  to  make  their  assessments  as  quickly  as  possible.  Therefore, 
perceived  time  pressure  may  account  for  some  of  the  errors.  Another  explanation  is  that  some  of 
these  symbols  were  similar  and  may  have  been  difficult  to  distinguish.  For  example,  the 
helicopter  and  rotary-wing  UAV  symbols  were  similar,  as  were  the  heavy  and  bomber  aircraft 
(both  with  wide  wingspans).  A  third  explanation  is  that  participants  received  limited  training,  so 
error  rates  are  likely  higher  than  would  be  seen  with  experienced  and  well-trained  operators. 


Table  2:  Symbology  Error  Rates 


Errors  (per  screen-shot) 


Cruise  missile 

0.017 

Ownship 

0.062 

Low  tech  fighter 

0.067 

Neutral 

0.067 

Unknown  class 

0.068 

Foe 

0.078 

Unknown 

0.078 

Friendly 

0.094 

High  tech  fighter 

0.151 

UAV  rotary-wing 

0.156 

Bomber 

0.164 

UAV  fixed-wing 

0.170 

Heavy 

0.211 

Helicopter 

0.289 

Together,  the  results  from  the  latency  analysis  and  the  error  analysis  indicate  that  participants 
were  able  to  recognize  and  categorize  the  new  UAV  symbols  with  similar  speed  and  accuracy  as 
compared  to  existing  MIL-STD-1787C  symbols. 

SITUATION  AWARENESS  ANALYSIS 


During  this  second  phase  of  the  evaluation,  participants  monitored  prerecorded  videos  of 
Open  UMI  scenarios  and  monitored  when  ownship  UAVs  entered  and  exited  restricted  airspace. 
They  also  monitored  when  alerts  were  presented  and  removed  for  the  ownship  UAVs.  The  alerts 
were  presented  as  small  symbols  tagged  to  an  air  vehicle  indicating  problems  with  engines, 
communications,  payloads,  etc.  (see  figure  2);  however,  participants  were  not  required  to 
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differentiate  between  the  various  types  of  alerts.  Rather,  they  simply  had  to  note  when  an  alert 
turned  on  and  turned  off. 

When  participants  detected  any  of  the  four  types  of  events,  they  were  required  to  verbally 
call  out  the  event  they  had  detected.  Each  participant  monitored  three  scenarios.  Each  scenario 
was  8  min  in  length.  One  scenario  was  sparsely  populated  with  two  ownship  UAVs  and,  on 
average,  seven  other  air  vehicles.  A  moderately  populated  scenario  had  3  ownship  UAVs  and,  on 
average,  11  other  air  vehicles.  A  densely  populated  scenario  had  5  ownship  UAVs  and,  on 
average,  18  other  air  vehicles.  The  total  number  of  UAV  events  ranged  from  31  in  the  sparsely 
populated  scenario  to  66  in  the  densely  populated  scenario. 

Measures  of  situation  awareness  included  the  latency  to  respond  and  correct  identification 
of  the  prescribed  events.  A  pilot  study  involving  tracking  non-UAV  MIL-STD-1787C  air 
vehicles  was  used  to  determine  a  performance  benchmark  from  which  success  or  failure  could  be 
measured  in  the  present  study.  The  pilot  study  involved  tracking  foe  aircraft  movements  into  and 
out  of  restricted  airspace,  as  well  as  into  and  out  of  the  on-screen  viewing  area  during  a  densely 
populated  scenario.  In  that  pilot  study,  the  average  time  to  detect  events  was  2.93  sec,  and  83% 
of  events  were  detected.  Using  those  numbers  as  guidelines,  a  response  latency  of  3  sec  or  less 
and  detection  of  at  least  85%  of  events  was  determined  to  be  a  reasonable  performance 
benchmark  for  this  type  of  task. 

The  situation  awareness  analysis  results  indicate  that  participants  performed  successfully 
relative  to  benchmark  levels  on  the  percent  of  events  detected  in  each  of  the  sparsely, 
moderately,  and  densely  populated  scenarios  (see  table  3).  Additionally,  participants  performed 
successfully  relative  to  benchmark  levels  for  latency  to  respond  in  each  of  the  three  scenarios 
(see  table  4).  Together,  these  results  suggest  that  the  newly  developed  UAS  symbology  allowed 
for  situation  awareness  slightly  better  than  that  found  with  the  existing  MIL-STD-1787C 
symbology  in  our  pilot  study. 

Table  3:  Percent  of  Events  Detected  by  UAV  Density 


Percent  of  Events  Detected 


Sparse 

92.26% 

Moderate 

90.73% 

Dense 

86.89% 

Table  4:  Detection  Latency  by  UAV  Density 


Latency  to  Respond  (sec) 


Sparse 

1.598 

Moderate 

1.847 

Dense 

2.059 
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CONCLUSIONS 

While  this  study  was  limited  in  scope,  the  results  provide  a  clear  indication  that  new  UAS 
symbols  can  be  successfully  developed  and  incorporated  into  the  existing  MIL-STD-1787C 
symbology  set.  Additionally,  the  new  symbols  developed  in  this  study  could  provide  individual 
operators  with  the  basic  elements  necessary  for  rapid  identification  and  tracking  of  UAVs  with 
appropriate  levels  of  situation  awareness. 

While  this  evaluation  provided  evidence  of  basic  utility  and  usability  of  a  set  of  UAS 
symbols,  additional  evaluations  of  these  symbols  should  be  carried  out  using  actual  UAS 
operators.  Furthermore,  the  tasks  within  the  present  study  did  not  simulate  a  number  of  critical 
aspects  of  actual  UAS  operations.  A  more  realistic  simulation  should  be  used  for  further 
evaluations  of  this  particular  symbol  set. 

Given  the  aforementioned  limitations,  this  study  represents  a  productive  step  in  the 
development  and  implementation  of  a  standard  UAS  symbology  set. 
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